Learning For Human Activity Recognition Research Articles

Self-supervised learning for human activity recognition using 700,000 person-days of wearable data

Accurate physical activity monitoring is essential to understand the impact of physical activity on one’s physical health and overall well-being. However, advances in human activity recognition algorithms have been constrained by the limited availability of large labelled datasets. This study aims to leverage recent advances in self-supervised learning to exploit the large-scale UK Biobank accelerometer dataset—a 700,000 person-days unlabelled dataset—in order to build models with vastly improved generalisability and accuracy. Our resulting models consistently outperform strong baselines across eight benchmark datasets, with an F1 relative improvement of 2.5–130.9% (median 24.4%). More importantly, in contrast to previous reports, our results generalise across external datasets, cohorts, living environments, and sensor devices. Our open-sourced pre-trained models will be valuable in domains with limited labelled data or where good sampling coverage (across devices, populations, and activities) is hard to achieve.

Semi-supervised Ensemble Learning for Human Activity Recognition in Casas Kyoto Dataset.

—The automatic identification of human physical activities, commonly referred to as Human Activity Recognition (HAR), has garnered significant interest and application across various sectors, including entertainment, sports, and notably health. Within the realm of health, a myriad of applications exists, contingent upon the nature of experimentation, the activities under scrutiny, and the methodology employed for data and information acquisition. This diversity opens doors to multifaceted applications, including support for the well-being and safeguarding of elderly individuals afflicted with neurodegenerative diseases, especially in the context of smart homes. Within the existing literature, a multitude of datasets from both indoor and outdoor environments have surfaced, significantly contributing to the activity identification processes. One prominent dataset, the CASAS project developed by Washington State University (WSU) University, encompasses experiments conducted in indoor settings. This dataset facilitates the identification of a range of activities, such as cleaning, cooking, eating, washing hands, and even making phone calls. This article introduces a model founded on the principles of Semi-supervised Ensemble Learning, enabling the harnessing of the potential inherent in distance-based clustering analysis. This technique aids in the identification of distinct clusters, each encapsulating unique activity characteristics. These clusters serve as pivotal inputs for the subsequent classification process, which leverages supervised techniques. The outcomes of this approach exhibit great promise, as evidenced by the quality metrics' analysis, showcasing favorable results compared to the existing state-of-the-art methods. This integrated framework not only contributes to the field of HAR but also holds immense potential for enhancing the capabilities of smart homes and related applications.

Robust Feature Representation Using Multi-Task Learning for Human Activity Recognition.

Learning underlying patterns from sensory data is crucial in the Human Activity Recognition (HAR) task to avoid poor generalization when coping with unseen data. A key solution to such an issue is representation learning, which becomes essential when input signals contain activities with similar patterns or when patterns generated by different subjects for the same activity vary. To address these issues, we seek a solution to increase generalization by learning the underlying factors of each sensor signal. We develop a novel multi-channel asymmetric auto-encoder to recreate input signals precisely and extract indicative unsupervised futures. Further, we investigate the role of various activation functions in signal reconstruction to ensure the model preserves the patterns of each activity in the output. Our main contribution is that we propose a multi-task learning model to enhance representation learning through shared layers between signal reconstruction and the HAR task to improve the robustness of the model in coping with users not included in the training phase. The proposed model learns shared features between different tasks that are indeed the underlying factors of each input signal. We validate our multi-task learning model using several publicly available HAR datasets, UCI-HAR, MHealth, PAMAP2, and USC-HAD, and an in-house alpine skiing dataset collected in the wild, where our model achieved 99%, 99%, 95%, 88%, and 92% accuracy. Our proposed method shows consistent performance and good generalization on all the datasets compared to the state of the art.

Self-attention deep ConvLSTM with sparse-learned channel dependencies for wearable sensor-based human activity recognition

In this study, we propose a novel deep-learning architecture with sparse learning for human activity recognition. The proposed model contains 1D CNNs and LSTM layers with a self-attention mechanism to enhance a substantial number of time points in time-series data for human activity recognition systems. Based on the recent success of squeeze-and-excite (SE) networks, the proposed deep learning model utilizes the SE module to enhance channel-wise interdependencies, which in turn leads to a boost in performance. In addition, we utilized sparse learning to retrain only weak nodes and freeze stronger nodes in a fully connected layer prior to classification layer. Furthermore, we utilized an entropy-inspired formula to find sparsely located weaker nodes and validated our model on various datasets, including Opportunity, UCI-HAR, and WISDM. Herein, we present an extensive analysis and survey of state-of-the-art studies, in addition to our proposed research. For a fair comparison, we evaluated our deep learning architecture using various performance metrics and achieved better results; the proposed model outperformed state-of-the-art algorithms for human activity recognition.

Hydra: Hybrid-model federated learning for human activity recognition on heterogeneous devices

Federated Learning (FL) has recently received extensive attention in enabling privacy-preserving edge AI services for Human Activity Recognition (HAR). However, users’ mobile and wearable devices in the HAR scenario usually possess dramatically different computing capability and diverse data distributions, making it very challenging for such heterogeneous HAR devices to conduct effective collaborative training (co-training) with the traditional FL schemes. To address this issue, we present Hydra, a Hybrid-model federated learning mechanism that facilitates the co-training among heterogeneous devices by allowing them to train models that well fit their own computing capability. Specifically, Hydra leverages BranchyNet to design a large-small global hybrid-model and enables heterogeneous devices to train the proper parts of the model tailored to their computing capability. Hydra drives co-training among the devices and clusters them based on model similarity to mitigate the impact of HAR data heterogeneity on model accuracy. In order to deal with the issue that large model may lack sufficient training data due to the limited number of high-performance devices in FL, we introduce a pairing scheme between high and low performance devices for effective co-training, and further propose sample selection approach to select more valuable samples to participate in co-training. We then formulate a constrained co-training problem within a cluster that is proved to be NP-hard and devise a fast greedy-based heuristic algorithm to solve it. In addition, to address the low accuracy of small models, we also propose a Large-to-Small knowledge distillation algorithm for resource-constrained devices to optimize the efficiency of transferring knowledge from large models to small models. We conduct extensive experiments on three HAR datasets and the experimental results demonstrate the superior performance of Hydra for achieving outstanding model accuracy improvement compared with other state-of-the-art schemes.

ProtoHAR: Prototype Guided Personalized Federated Learning for Human Activity Recognition.

Federated Learning (FL) has recently attracted great interest in sensor-based human activity recognition (HAR) tasks. However, in real-world environment, sensor data on devices is non-independently and identically distributed (Non-IID), e.g., activity data recorded by most devices is sparse, and sensor data distribution for each client may be inconsistent. As a result, the traditional FL methods in the heterogeneous environment may incur a drifted global model that causes slow convergence and a heavy communication burden. Although some FL methods are gradually being applied to HAR, they are designed for overly ideal scenarios and do not address such Non-IID problem in the real-world setting. It is still a question whether they can be applied to cross-device FL. To tackle this challenge, we propose ProtoHAR, a prototype-guided FL framework for HAR, which aims to decouple the representation and classifier in the heterogeneous FL setting efficiently. It leverages the global prototype to correct the activity feature representation to make the prototype knowledge flow among clients without leaking privacy while solving a better classifier to avoid excessive drift of the local model in personalized training. Extensive experiments are conducted on four publicly available datasets: USC-HAD, UNIMIB-SHAR, PAMAP2, and HARBOX, which are collected in both controlled environments and real-world scenarios. The results show that compared with the state-of-the-art FL algorithms, ProtoHAR achieves the best performance and faster convergence speed in HAR datasets.

Online continual learning for human activity recognition

Sensor-based human activity recognition (HAR), with the ability to recognise human activities from wearable or embedded sensors, has been playing an important role in many applications including personal health monitoring, smart home, and manufacturing. The real-world, long-term deployment of these HAR systems drives a critical research question: how to evolve the HAR model automatically over time to accommodate changes in an environment or activity patterns. This paper presents an online continual learning (OCL) scenario for HAR, where sensor data arrives in a streaming manner which contains unlabelled samples from already learnt activities or new activities. We propose a technique, OCL-HAR, making a real-time prediction on the streaming sensor data while at the same time discovering and learning new activities. We have empirically evaluated OCL-HAR on four third-party, publicly available HAR datasets. Our results have shown that this OCL scenario is challenging to state-of-the-art continual learning techniques that have significantly underperformed. Our technique OCL-HAR has consistently outperformed them in all experiment setups, leading up to 0.17 and 0.23 improvements in micro and macro F1 scores.

Deep Learning for Human Activity Recognition on 3D Human Skeleton: Survey and Comparative Study.

Human activity recognition (HAR) is an important research problem in computer vision. This problem is widely applied to building applications in human-machine interactions, monitoring, etc. Especially, HAR based on the human skeleton creates intuitive applications. Therefore, determining the current results of these studies is very important in selecting solutions and developing commercial products. In this paper, we perform a full survey on using deep learning to recognize human activity based on three-dimensional (3D) human skeleton data as input. Our research is based on four types of deep learning networks for activity recognition based on extracted feature vectors: Recurrent Neural Network (RNN) using extracted activity sequence features; Convolutional Neural Network (CNN) uses feature vectors extracted based on the projection of the skeleton into the image space; Graph Convolution Network (GCN) uses features extracted from the skeleton graph and the temporal-spatial function of the skeleton; Hybrid Deep Neural Network (Hybrid-DNN) uses many other types of features in combination. Our survey research is fully implemented from models, databases, metrics, and results from 2019 to March 2023, and they are presented in ascending order of time. In particular, we also carried out a comparative study on HAR based on a 3D human skeleton on the KLHA3D 102 and KLYOGA3D datasets. At the same time, we performed analysis and discussed the obtained results when applying CNN-based, GCN-based, and Hybrid-DNN-based deep learning networks.

Transfer Learning for Human Activity Recognition Using Representational Analysis of Neural Networks

Human activity recognition (HAR) has increased in recent years due to its applications in mobile health monitoring, activity recognition, and patient rehabilitation. The typical approach is training a HAR classifier offline with known users and then using the same classifier for new users. However, the accuracy for new users can be low with this approach if their activity patterns are different than those in the training data. At the same time, training from scratch for new users is not feasible for mobile applications due to the high computational cost and training time. To address this issue, we propose a HAR transfer learning framework with two components. First, a representational analysis reveals common features that can transfer across users and user-specific features that need to be customized. Using this insight, we transfer the reusable portion of the offline classifier to new users and fine-tune only the rest. Our experiments with five datasets show up to 43% accuracy improvement and 66% training time reduction when compared to the baseline without using transfer learning. Furthermore, measurements on the hardware platform reveal that the power and energy consumption decreased by 43% and 68%, respectively, while achieving the same or higher accuracy as training from scratch. Our code is released for reproducibility. 1

Sensor Data Augmentation by Resampling in Contrastive Learning for Human Activity Recognition

While deep learning models have contributed to the advancement of sensor-based human activity recognition (HAR), it usually requires large amounts of annotated sensor data to extract robust features. To alleviate the limitations of data annotation, contrastive learning has been applied to sensor-based HAR. One of the essential factors of contrastive learning is data augmentation, significantly impacting the performance of pretraining. However, current popular augmentation methods do not achieve competitive performance in contrastive learning for sensor-based HAR. Motivated by this issue, we propose a new sensor data augmentation method by resampling, which introduces variable domain information and simulates realistic activity data by varying the sampling frequency to maximize the coverage of the sampling space. The resampling augmentation method was evaluated in supervised learning and contrastive learning [SimCLR for HAR (SimCLRHAR) and MoCo for HAR (MoCoHAR)]. In the experiment, we use four datasets, UCI-HAR, MotionSense, USC-HAD, and MobiAct, using the mean F1-score as the evaluation metric for downstream tasks. The experimental results show that the resampling data augmentation outperforms all state-of-the-art augmentation methods in supervised learning and contrastive learning with a small amount of labeled data. The results also demonstrate that not all data augmentation methods have positive effects in contrastive learning frameworks.

TASKED: Transformer-based Adversarial learning for human activity recognition using wearable sensors via Self-KnowledgE Distillation

Wearable sensor-based human activity recognition (HAR) has emerged as a principal research area and is utilized in a variety of applications. Recently, deep learning-based methods have achieved significant improvement in the HAR field with the development of human–computer interaction applications. However, most of them are limited to operating in a local neighborhood in the process of a standard convolution neural network, and correlations between different sensors on body positions are ignored. In addition, even though several recent existing works considered the correlations between different sensor positions, they still face significant challenging problems with performance degradation due to large gaps in the distribution of training and test data, and behavioral differences between subjects. In this work, we propose a novel Transformer-based Adversarial learning framework for human activity recognition using wearable sensors via Self-KnowledgE Distillation (TASKED), that accounts for individual sensor orientations and spatial and temporal features. The proposed method is capable of learning cross-domain embedding feature representations from multiple subjects datasets using adversarial learning and the maximum mean discrepancy (MMD) regularization to align the data distribution over multiple domains. In the proposed method, we adopt the teacher-free self-knowledge distillation to improve the stability of the training procedure and the performance of human activity recognition. Experimental results show that TASKED not only outperforms state-of-the-art methods on the four real-world public HAR datasets (alone or combined) but also improves the subject generalization effectively.

Augmented Adversarial Learning for Human Activity Recognition with Partial Sensor Sets

Human activity recognition (HAR) plays an important role in a wide range of applications, such as health monitoring and gaming. Inertial sensors attached to body segments constitute a critical sensing system for HAR. Diverse inertial sensor datasets for HAR have been released with the intention of attracting collective efforts and saving the data collection burden. However, these datasets are heterogeneous in terms of subjects and sensor positions. The coupling of these two factors makes it hard to generalize the model to a new application scenario, where there are unseen subjects and new sensor position combinations. In this paper, we design a framework to combine heterogeneous data to learn a general representation for HAR, so that it can work for new applications. We propose an Augmented Adversarial Learning framework for HAR (AALH) to learn generalizable representations to deal with diverse combinations of sensor positions and subject discrepancies. We train an adversarial neural network to map various sensor sets' data into a common latent representation space which is domain-invariant and class-discriminative. We enrich the latent representation space by a hybrid missing strategy and complement each subject domain with a multi-domain mixup method, and they significantly improve model generalization. Experiment results on two HAR datasets demonstrate that the proposed method significantly outperforms previous methods on unseen subjects and new sensor position combinations.

Evolutionary Dual-Ensemble Class Imbalance Learning for Human Activity Recognition

Human activity recognition is an imbalance classification problem in essence since various human actions may occur at different frequencies. Traditional ensemble class imbalance learning methods integrate resampling technique with multi-classifier models, which obtain better generalization than single ones. However, the number of base classifiers is often determined in advance, resulting in the redundant structure of a multi-classifier and larger computational cost. Moreover, various combinations of base classifiers may have similar recognition accuracy, forming a multimodel optimization problem. To address the issue, a dual-ensemble class imbalance learning method is presented, in which two ensemble learning models are nested each other. For each sub-dataset, three heterogeneous sub-classifiers compose of internal ensemble learning model, and the one with the highest recognition accuracy is selected as base classifier. In external ensemble learning model, an optimal combination that contains the smallest number of base classifiers and identifies human actions most exactly is found by multimodal evolutionary algorithms. Based on seven imbalanced activity datasets derived from localization data for person activity, statistical experimental results show that the proposed evolutionary dual-ensemble class imbalance learning method provides the simplest ensemble structure with the best accuracy for imbalanced data that outperforms other five widely-used ensemble classification methods on all metrics.

Gradient Descent Machine Learning with Equivalency Testing for Non-Subject Dependent Applications in Human Activity Recognition

INTRODUCTION: A solution to subject-independent HAR prediction through machine learning classification algorithms using statistical equivalency for comparative analysis between independent groups with non-subject training dependencies.OBJECTIVES: To indicate that the multinomial predictive classification model that was trained and optimized on the one-subject control group is at least partially extensible to multiple independent experiment groups for at least one activity class.METHODS: Gradient boosted machine multinomial classification algorithm is trained on a single individual with the classifier trained on all activity classes as a multinomial classification problem.RESULTS: Levene-Wellek-Welch (LWW) Statistic calculated as 0.021, with a Critical Value for LWW of 0.026, using an alpha of 0.05.CONCLUSION: Confirmed falsifiability that incorporates reproducible methods into the quasi-experiment design applied to the field of machine learning for human activity recognition.

Vision Transformer and Deep Sequence Learning for Human Activity Recognition in Surveillance Videos

Human Activity Recognition is an active research area with several Convolutional Neural Network (CNN) based features extraction and classification methods employed for surveillance and other applications. However, accurate identification of HAR from a sequence of frames is a challenging task due to cluttered background, different viewpoints, low resolution, and partial occlusion. Current CNN-based techniques use large-scale computational classifiers along with convolutional operators having local receptive fields, limiting their performance to capture long-range temporal information. Therefore, in this work, we introduce a convolution-free approach for accurate HAR, which overcomes the above-mentioned problems and accurately encodes relative spatial information. In the proposed framework, the frame-level features are extracted via pretrained Vision Transformer; next, these features are passed to multilayer long short-term memory to capture the long-range dependencies of the actions in the surveillance videos. To validate the performance of the proposed framework, we carried out extensive experiments on UCF50 and HMDB51 benchmark HAR datasets and improved accuracy by 0.944% and 1.414%, respectively, when compared to state-of-the-art deep models.

Impact of Wireless Sensor Data Mining with Hybrid Deep Learning for Human Activity Recognition

Human activity recognition is a time series classification problem that is difficult to solve (HAR). Traditional signal processing approaches and domain expertise are necessary to appropriately create features from raw data and fit a machine learning model for predicting a person’s movement. This work aims to demonstrate how a hybrid deep learning model may be used to recognize human behavior. Deep learning methodologies such as convolutional neural networks and recurrent neural networks will extract the features and achieve the classification goal. The suggested model has used wireless sensor data mining datasets to predict human activity. The model’s performance has been assessed using the confusion matrix, accuracy, training loss, and testing loss. Thus, the model has achieved greater than 96% accuracy, superior to other state-of-the-art algorithms in this field.

Feature matching and instance reweighting with transfer learning for human activity recognition using smartphone

Feature matching and instance reweighting with transfer learning for human activity recognition using smartphone

Ultra-Wideband Radar-Based Activity Recognition Using Deep Learning

With recent advances in the field of sensing, it has become possible to build better assistive technologies. This enables the strengthening of eldercare with regard to daily routines and the provision of personalised care to users. For instance, it is possible to detect a person’s behaviour based on wearable or ambient sensors; however, it is difficult for users to wear devices 24/7, as they would have to be recharged regularly because of their energy consumption. Similarly, although cameras have been widely used as ambient sensors, they carry the risk of breaching users’ privacy. This paper presents a novel sensing approach based on deep learning for human activity recognition using a non-wearable ultra-wideband (UWB) radar sensor. UWB sensors protect privacy better than RGB cameras because they do not collect visual data. In this study, UWB sensors were mounted on a mobile robot to monitor and observe subjects from a specific distance (namely, 1.5–2.0 m). Initially, data were collected in a lab environment for five different human activities. Subsequently, the data were used to train a model using the state-of-the-art deep learning approach, namely long short-term memory (LSTM). Conventional training approaches were also tested to validate the superiority of LSTM. As a UWB sensor collects many data points in a single frame, enhanced discriminant analysis was used to reduce the dimensions of the features through application of principal component analysis to the raw dataset, followed by linear discriminant analysis. The enhanced discriminant features were fed into the LSTMs. Finally, the trained model was tested using new inputs. The proposed LSTM-based activity recognition approach performed better than conventional approaches, with an accuracy of 99.6%. We applied 5-fold cross-validation to test our approach. We also validated our approach on publically available dataset. The proposed method can be applied in many prominent fields, including human–robot interaction for various practical applications, such as mobile robots for eldercare.

Deep Learning for Human Activity Recognition Based on Causality Feature Extraction

We propose a novel data-driven feature extraction approach based on direct causality and fuzzy temporal windows (FTWs) to improve the precision of human activity recognition and mitigate the problems of easily-confused activities and unlabeled data, which significantly degrade classification performance owing to the correlation of labeled data. In recognizing activities, the proposed approach not only considers the importance of oncoming short-term sensor data but also considers the continuity from past activities of the preceding long-term sensor data. In terms of the oncoming data, the causality feature is extracted using the direct transfer entropy to determine the unique pattern of an activity, which represents the quantified causal relationship between sensor activations. In terms of the preceding data, several hours of historical data are compressed to fuzzy features based on FTWs. Subsequently, the causality and fuzzy features are fused by matrix multiplication to express distinct features of activities. To effectively learn the spatiotemporal dependencies of the fused feature, deep long short-term memory (LSTM), two-dimensional convolutional neural network (2D-CNN), and hybrid models composed of a combination of LSTM and CNN were used. Leave-one-day-out cross-validation was performed based on the CASAS open datasets, including Aruba, Cairo, and Milan. The results showed that the macro-F1-scores were improved by 16.4, 37.5, and 18.5%, respectively, compared with those of the FTW-only environments. In addition, the proposed approach could improve the precision of activity recognition and mitigate the problems associated with the environments containing unlabeled data.

A Benchmark of Data Stream Classification for Human Activity Recognition on Connected Objects

This paper evaluates data stream classifiers from the perspective of connected devices, focusing on the use case of Human Activity Recognition. We measure both the classification performance and resource consumption (runtime, memory, and power) of five usual stream classification algorithms, implemented in a consistent library, and applied to two real human activity datasets and three synthetic datasets. Regarding classification performance, the results show the overall superiority of the Hoeffding Tree, the Mondrian forest, and the Naïve Bayes classifiers over the Feedforward Neural Network and the Micro Cluster Nearest Neighbor classifiers on four datasets out of six, including the real ones. In addition, the Hoeffding Tree and—to some extent—the Micro Cluster Nearest Neighbor, are the only classifiers that can recover from a concept drift. Overall, the three leading classifiers still perform substantially worse than an offline classifier on the real datasets. Regarding resource consumption, the Hoeffding Tree and the Mondrian forest are the most memory intensive and have the longest runtime; however, no difference in power consumption is found between classifiers. We conclude that stream learning for Human Activity Recognition on connected objects is challenged by two factors which could lead to interesting future work: a high memory consumption and low F1 scores overall.